@Sujanadh sorry to distract from the fmtm-splitter work, but could you please take a look at this PR tomorrow?

I was coding into the night, so no doubt have missed something. Please check everything works as intended / looks good to you 😄

This will finalise the work on injecting XLSForm fields and allow #1653 to be easily solved.

Thank you!

sorry i didn't see the PR , i will test it

Encountered few issues while testing will post it here briefly.

Thanks! There are definitely some bugs to find 😆

I didn't actually get to test updating an existing form either, so that probably needs some work!

HTTPStatus.UNPROCESSABLE_ENTITY: XLSForm is invalid: There are two sections with the name survey_questions.
yes while appending there is a problem since we need to get common fields and then append likewise

I used default form here.

when trying to check custom form also there is issue

{
  "detail": "XLSForm is invalid: There has been a problem trying to replace ${existing} with the XPath to the survey element named 'existing'. There is no survey element with this name."
}

this was the particular reason that i had to first append mandatory fields before testing for validation.

oh actually my bad , i should debug on for custom form , it works fine on custom form

i don't think we need to validate it again in generate-project-date since we append and validate using validate-form endpoint then return valid appended form which is used while generating project data. For default form we don't need to do anything , instead we can change xls form to allow to create new entity with new geopoint.

Thanks for the tests!

The issue about existing is from the old XLSForm that has calculated fields based on this. As you say, using a custom form without this logic works fine.

Agree we could validate once only!
The reason I left it in is because the API is independent from the frontend.

While it's quite likely our frontend is the only user of the API, calling validate-form prior to generating project files, we can't guarantee it. So validating again is just an addition check, without much performance penalty - what do you think?

i don't think we need to validate it again in generate-project-date since we append and validate using validate-form endpoint then return valid appended form which is used while generating project data. For default form we don't need to do anything , instead we can change xls form to allow to create new entity with new geopoint.

Regarding the default forms, I already updated the forms to remove the verification fields (so they are injected).

I will update them again now to remove the FMTM mandatory fields entirely, and have them injected the same way as a custom form.

This should improve maintainability, as we don't need to update the entire XLSForm library when we change something in the mandatory fields 😄

Related issue I just made: hotosm/osm-fieldwork#300

HTTPStatus.UNPROCESSABLE_ENTITY: XLSForm is invalid: There are two sections with the name survey_questions. yes while appending there is a problem since we need to get common fields and then append likewise

I used default form here.

This is a valid error though! We are having issues with the dataframe merging logic

Thanks for the tests!

The issue about existing is from the old XLSForm that has calculated fields based on this. As you say, using a custom form without this logic works fine.

Agree we could validate once only! The reason I left it in is because the API is independent from the frontend.

While it's quite likely our frontend is the only user of the API, calling validate-form prior to generating project files, we can't guarantee it. So validating again is just an addition check, without much performance penalty - what do you think?

Ah got it, you are right that makes sense.

Made some updates!

Hopefully fixed project creation and XLSForm updating 😄

I made some updates to osm-fieldwork for this, so the main things that need to be done:

• Do some tests all the way through to the ODK Collect usage, to check the logic all works as intended.
• Deploy this to staging for Monday.

I am merging this now so we can more easily test - we can make another PR if needed.